1. Field of the Invention
The present invention relates to an image pickup apparatus which offers high quality of images by using a technique referred to as xe2x80x9cpixel shiftxe2x80x9d.
2. Description of the Related Art
In recent years, there is an increasing demand for image inputting apparatuses, due to rapid spreading of use of personal computers (PCs).
Typically, scanners are used as the image inputting apparatuses for use in combination with PCs. Such scanners are basically intended to pickup image information written or printed on sheets of paper or the like, such as written or printed texts, photographs and so on.
Scanners employ line sensors as imaging means, and cover a wide range of resolution from 200 dpi (dots per inch) up to very high resolution of 1200 dpi.
In general, sufficient legibility and readability of 10-point fonts of a written or printed text is considered as criteria for the image resolution in practical applications. To meet this requirement, the image inputting apparatuses have to pickup and form images at a resolution which is at least 200 dpi. Thus, the scanners satisfy this requirement insofar as the image resolution is concerned.
In the meantime, image inputting apparatuses relying upon video techniques are also available, a typical example of which is a visualizer which also is known as xe2x80x9cdocument camera.xe2x80x9d Many visualizers have been proposed and are actually available as commercial products.
In most cases, a visualizer has an illuminated platen for carrying an original document, and a video camera which is supported on the platen and which picks up the image of the original document.
Such a visualizer employs an area sensor as the imaging means. The image resolution obtainable with this type of apparatus is limited by the resolution achievable by the video camera. In general, a video camera can provide only low levels of image resolution. For instance, the image resolution is about 67 dpi at the highest, when the video camera is used for reading a full A-4 size text which is laid with its longer side extending from left to right. More specifically, the pixel pitch along the longer side of the A-4 size text is calculated as 640/9.5=67.4 (dpi) at the aspect ratio of 1:1 when the video camera follows NTSC specifications.
Referring again to scanners, one of the problems encountered with this type of image inputting apparatus is excessively long scanning time. In fact, it takes about 10 to 20 seconds to scan a color image. Another problem is that the whole image, i.e., the overall framing, cannot be grasped until the entire original image is scanned.
Apparatuses having a preview mode has been proposed, in order to obviate the second-mentioned problem, i.e., inability to grasp the overall framing before completion of scanning. Such apparatuses, however, are not easy to operate, due to necessity of certain steps, including previewing and subsequent pointing of the scanning area by a pointing device such as a mouse.
A fundamental problem encountered with the use of scanners is that the scanners cannot pickup images of three-dimensional objects. In other words, the scanners are effectively used only for reading texts and pictures.
In contrast, video visualizers are easy to use and can pickup images of three-dimensional objects while offering easy framing, thus obviating the problems encountered with the scanners, but are still unsatisfactory as they cannot satisfy requirements in regard to image resolution.
It is true that video visualizers can provide only low levels of resolution, e.g., 67 dpi at the highest, failing to meet the demand for higher image quality, as described before. Such video visualizers, on the other hand, offer a remarkable advantage, by virtue of use of a zooming function. More specifically, the user can control and determine the framing in real time by changing the zooming ratio and moving the original document on the platen while monitoring the image through a display.
It is therefore highly desirable that a video visualizer has a high level of resolution that compares well with that achieved by scanners. Such a video visualizer will make it possible to obtain high-quality images not only from texts or pictures but also from three-dimensional objects.
One of the measures for implementing such an image pickup apparatus would be to increase the number of pixels in the area sensor used in a video visualizer. The area sensors used in existing video visualizers have 410,000 pixels in total, among which 380,000 are effective, in order to satisfy the requirement for resolution corresponding to 450 TV lines. In the meantime, high-quality image sensors intended for use in digital still cameras have been proposed in recent years. These sensors have very large numbers of pixels, e.g., 1,000,000 pixels, 1,300,000 pixels or 1,600,000 pixels. It might be possible to substitute such a high-quality image sensor for a conventional area sensor in a video visualizer, in order to achieve the desired high level of resolution.
Such high-quality image sensors, however, have large chip sizes and, hence, are generally expensive. An attempt to enhance the scale of integration through reduction of pixel size is not considered promising, because such higher scale of integration poses other problems such as impairment of S/N ratio due to reduction in the sensitivity.
Moreover, it is to be pointed out that these high-quality image sensors are still unsatisfactory because they cannot provide image resolution equivalent to that offered by scanners. More specifically, the resolution is as low as 100 dpi when the area sensor having 1,000,000 pixels is used for reading a text of A-4 size. Similarly, the resolution is still as low as 135 dpi, when the area sensor having 1,350,000 pixels is used for reading A-4 size text and, even when an area sensor having 1,600,000 pixels is used, the resolution obtainable is still 135 dpi after conversion into image having the aspect ratio of 1:1. Such resolutions are much lower than 200 dpi easily achievable by a scanner having comparatively low resolution.
Furthermore, the video visualizer incorporating such a high-quality image sensor is required to produce analog video output in accordance with, for example, NTSC specifications, in order to attain compatibility with existing video visualizers. The analog video output is preferably of an image quality on the order of 410,000 in terms of pixel number (380,000 effective pixels). However, the above-mentioned high-quality image sensor, due to its specific pixel arrangement, lacks any output mode which would enable conversion of its output to the 410,000-pixel video rate. In order to implement such an output mode, it is necessary to additionally incorporate a scan rate converting circuit, with the result that the production cost is raised undesirably.
In addition, as is well known, the scan rate conversion itself is a factor which degrades the image quality, because it produces unnatural minute steps or jags at the edges of image. Therefore, even when the scan rate for the output of the high-quality image sensor could be converted to that corresponding to 410,000 pixels, the image quality obtained with such a high-quality image sensor is inevitably inferior to the quality of image derived from the conventional sensor having 410,000 pixels, due to the presence of minute steps or jags on the edges of images.
In order to obviate this problem, a method generally referred to as xe2x80x9cpixel shiftxe2x80x9d has been proposed in which the position of the imaging device relative to the incident light is shifted stepwise at a pitch which is a fraction (1/N) of the pixel number, thereby improving the image resolution.
More specifically, the pixel shift can be realized by using a plane-parallel light transmissive glass plate disposed in the optical path at a location between the image pickup lens and the imaging device. The light-transmissive glass plate is rotated by a predetermined amount, so that the light rays carrying optical image information, which otherwise would impinge upon insensitive zones between the pixels, can be received by the pixels, thus enhancing the image resolution.
The pixel shift can be implemented also by effecting a minute shift of the imaging device itself. This method, however, is not practical because of the difficulty encountered in achieving required level of precision of the position control.
The use of the pixel shift technique makes it possible to pickup images at high level of quality without requiring greater number of pixels, by virtue of the effect produced by the plane-parallel glass plate. This technique, therefore, has been appreciated as being a very effective measure for achieving high image quality.
The pixel-shift technique also makes it possible to easily obtain analog video signal simultaneously by using, as the imaging device, a 410,000-pixel video sensor.
However, the following problems are encountered in implementing a visualizer having image resolution on the order of 200 dpi by using the conventional pixel-shift technique.
Assuming the simplest form of the pixel shift in which the pitch of the shift is xc2xd the pixel pitch, the resolution cannot be increased beyond double of the original resolution. For instance, when the visualizer is used for reading full text on an A-4 paper which is laid with its longer side extending from the right to the left, the maximum resolution is 135 dpi (67xc3x972).
This level of resolution is insufficient for picking up images of 10-point fonts on a paper sheet. In addition, picking up of colored image requires an image memory having 16 frames, which raises the cost uneconomically.
Furthermore, color signal processings such as white-balance processing has to be conducted on an externally connected PC, requiring a long processing time which is usually from 10 to 20 seconds.
In addition, a computation is necessary for generating monochromatic signal, because of the use of primary color filters, regardless of whether the filters are pure color filters or complementary color filters. Such computation also has to be conducted by the external PC, further prolonging the processing time.
The minimum required resolution level of 200 dpi is obtainable if the pitch of the pixel shift is set to be ⅓ pixel pitch. In such a case, however, an image memory having 36 frames is required, resulting in a rise of the cost as in the case of the pixel shift by xc2xd pixel pitch. Processing time is prolonged also in this case, due to necessity of color signal processing such as white balancing.
Furthermore, regardless of the pitch of the pixel shift, it is necessary to employ a signal processing circuit which processes video signals obtained as a result of the pixel-shifting operation and which is incompatible with ordinary signal processing circuit. Thus, an exclusive signal processing circuit is required or the external PC is burdened if the signal processing is performed on the PC.
As an example of diversification of functions of visualizers, an apparatus has been proposed in which the posture of the image pickup unit is variable to enable pickup of image of a moving object rather than original texts. This type of apparatus must have separate and independent signal processing systems, one for picking up moving image and one for picking up still images at high resolution through pixel shifting. This type of apparatus, therefore, is not easy to implement, due to reasons such as large scale of system structure, rise of the cost, long processing time, and so forth.
Accordingly, an object of the present invention is to provide an image pickup apparatus which offers, by virtue of use of a pixel-shift technique, various advantages such as high quality of output image, high processing efficiency, high speed of operation and high efficiency of use of memory.
Another object of the present invention is to provide an image pickup apparatus relying upon a pixel-shift technique, improved to enable an existing camera processing circuit to process image data picked up through the pixel-shifting operation, thereby offering higher signal processing efficiency and higher efficiency of use of memory, as well as higher operation speed.
To these ends, according to one aspect of the present invention, there is provided a image pickup apparatus, comprising: pixel-shifting means for shifting a light beam impinging upon the imaging surface of an imaging device at a shifting pitch corresponding to a predetermined number of pixels; an image memory for storing pixel data obtained at each of the positions to which the light beam has been shifted by the pixel-shifting means; and a memory controller for causing the image memory to store the pixel data in a spatial arrangement equivalent to the pixel data obtained by an imaginary imaging device having pixels of a number N (N being an integer) times as large as that of the imaging device in each of the line and column directions, and for dividing the stored pixel data into frames to enable reading of the stored pixel data in a plurality of times on a frame basis.
According to another aspect of the present invention, there is provided an image pickup apparatus, comprising: an imaging device which enables reading of pixel data both in a frame reading mode in which data of all the pixels are read and a field reading mode in which pixel data from a plurality of lines of pixels in a mixed manner; a camera process circuit for processing pixel data read from the imaging device in the plural-line-mixed field reading mode; pixel-shifting means for shifting a light beam impinging upon the imaging surface of an imaging device at a shifting pitch corresponding to a predetermined number of pixels; an image memory for storing pixel data read from the imaging device; a memory controller for causing, when the pixel data read from the imaging device in the frame reading mode are written in the image memory, the image memory to store the pixel data in a spatial arrangement equivalent to the pixel data obtained by an imaginary imaging device having pixels of a number N (N being an integer) times as large as that of the imaging device in each of the line and column directions, and for dividing the stored pixel data into frames to enable reading of the stored pixel data in a plurality of times on a frame basis; and camera processing means for processing field-read pixel data obtained through a computation for mixing pixel data of a plurality of lines performed on the pixel data read by the memory controller from the image memory.
The invention also provides in its further aspect an image pickup apparatus, comprising: an imaging device provided with a color filter having a regular arrangement formed by repetition of an 8-pixel unit block having two pixels in the line direction and 4 pixels in the column direction; pixel-shifting means operative to shift a light beam impinging upon the imaging surface of the imaging device by an amount corresponding to ⅔ pixel pitch both in plus and minus directions: A/D converter means for converting the pixel data accumulated on the imaging device into digital data; an image memory capable of storing at least 9 frames of digital pixel data outputted from the A/D converter means; a memory controller for arranging the pixel data to be written in the image memory in a spatial arrangement equivalent to the pixel data obtained by an imaginary imaging device having pixels of a number three times as large as that of the imaging device in each of line and column directions; an adder circuit which performs a computation for mixing pixel data of a plurality of pixel lines read from the image memory; and camera process means for processing the plural-line-mixed field-read pixel data outputted from the adder circuit.
A third object of the present invention is to provide an image pickup apparatus incorporating a low-pass filter control suitable for use in the image pickup operation relying on pixel-shifting technique.